Methods and systems for adaptive radiotherapy treatment planning using deep learning engines

ABSTRACT

Example methods for adaptive radiotherapy treatment planning using deep learning engines are provided. One example method may comprise obtaining treatment image data associated with a first imaging modality and planning image data associated with a second imaging modality. The treatment image data may be acquired during a treatment phase of a patient. Also, planning image data associated with a second imaging modality may be acquired prior to the treatment phase to generate a treatment plan for the patient. The method may also comprise: in response to determination that an update of the treatment plan is required, processing, using the deep learning engine, the treatment image data and the planning image data to generate output data for updating the treatment plan.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation under 35 U.S.C. § 120 of U.S.patent application Ser. No. 16/145,673, filed Sep. 28, 2018, which isrelated in subject matter to U.S. patent application Ser. Nos.16/145,461 and 16/145,606 (now U.S. Pat. No. 10,984,902 B2). The U.S.patent applications, including any appendices or attachments thereof,are incorporated by reference herein in their entirety.

BACKGROUND

Unless otherwise indicated herein, the approaches described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Radiotherapy is an important part of a treatment for reducing oreliminating unwanted tumors from patients. Unfortunately, appliedradiation does not inherently discriminate between an unwanted tumor andany proximal healthy structures such as organs, etc. This necessitatescareful administration to restrict the radiation to the tumor (i.e.,target). Ideally, the goal is to deliver a lethal or curative radiationdose to the tumor, while maintaining an acceptable dose level in theproximal healthy structures. However, to achieve this goal, conventionalradiotherapy treatment planning and/or adaptive radiotherapy treatmentplanning may be time and labor intensive.

SUMMARY

According to a first aspect of the present disclosure, example methodsand systems for radiotherapy treatment planning using a deep learningengine are provided. Various examples will be discussed using FIG. 1 toFIG. 5 . The deep learning engine may include at least a firstprocessing pathway, a second processing pathway and a third processingpathway. One example method may comprise obtaining first image dataassociated with a patient; generating first feature data by processingthe first image data associated with a first resolution level using thefirst processing pathway; generating second feature data by processingsecond image data associated with a second resolution level using thesecond processing pathway; and generating third feature data byprocessing third image data associated with a third resolution levelusing the third processing pathway. The example method may also comprisegenerating a first combined set of feature data associated with thesecond resolution level based on the second feature data and the thirdfeature data, and a second combined set of feature data associated withthe first resolution level based on the first feature data and the firstcombined set. Further, the example method may comprise generating outputdata associated with radiotherapy treatment of the patient. For example,the output data may include at least one of the following: structuredata associated with the patient, dose data associated with the patient,and treatment delivery data for a treatment delivery system.

According to a second aspect of the present disclosure, example methodsand systems for adaptive radiotherapy treatment planning using a deeplearning engine are provided. Various examples will be discussed usingFIG. 6 and FIG. 7 . One example method may comprise obtaining treatmentimage data associated with a first imaging modality. The treatment imagedata may be acquired during a treatment phase of a patient. Also,planning image data associated with a second imaging modality may beacquired prior to the treatment phase to generate a treatment plan forthe patient. The method may also comprise: in response to determinationthat an update of the treatment plan is required, transforming thetreatment image data associated with the first imaging modality togenerate transformed image data associated with the second imagingmodality. The method may further comprise: processing, using the deeplearning engine, the transformed image data to generate output data forupdating the treatment plan. For example, the output data may be atleast one of the following: structure data associated with the patient,dose data associated with the patient, and treatment delivery data for atreatment delivery system.

According to a third aspect of the present disclosure, example methodsand systems for adaptive radiotherapy treatment planning using a deeplearning engine are provided. Various examples will be discussed usingFIG. 6 and FIG. 8 . One example method may comprise obtaining treatmentimage data associated with a first imaging modality and planning imagedata associated with a second imaging modality. The planning image datamay be acquired prior to the treatment phase to generate a treatmentplan for the patient. The method may also comprise: in response todetermination that an update of the treatment plan is required,processing, using the deep learning engine, the treatment image data andthe planning image data to generate output data for updating thetreatment plan. For example, the output data may be at least one of thefollowing: structure data associated with the patient, dose dataassociated with the patient, and treatment delivery data for a treatmentdelivery system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating an example process flow forradiotherapy treatment;

FIG. 2 is a schematic diagram illustrating an example deep learningengine with multiple processing pathways to perform radiotherapytreatment planning;

FIG. 3 is a flowchart of an example process for a computer system toperform radiotherapy treatment planning using a deep learning engine;

FIG. 4 is a schematic diagram illustrating an example deep learningengine to perform automatic segmentation of image data for radiotherapytreatment planning;

FIG. 5 is a schematic diagram illustrating an example deep learningengine to perform dose prediction for radiotherapy treatment planning;

FIG. 6 is schematic diagram illustrating an example process flow for acomputer system to perform adaptive radiotherapy treatment (ART)planning using a deep learning engine;

FIG. 7 is a schematic diagram illustrating a first example approach forART planning according to the example in FIG. 6 ;

FIG. 8 is a schematic diagram illustrating a second example approach forART planning according to the example in FIG. 6 ;

FIG. 9 is schematic diagram illustrating an example treatment plangenerated or improved based on output data in the examples in FIG. 1 toFIG. 8 ; and

FIG. 10 is a schematic diagram of an example computer system to performradiotherapy treatment planning and/or adaptive radiotherapy treatmentplanning.

DETAILED DESCRIPTION

The technical details set forth in the following description enable aperson skilled in the art to implement one or more embodiments of thepresent disclosure.

FIG. 1 is a schematic diagram illustrating example process flow 100 forradiotherapy treatment. Example process 100 may include one or moreoperations, functions, or actions illustrated by one or more blocks. Thevarious blocks may be combined into fewer blocks, divided intoadditional blocks, and/or eliminated based upon the desiredimplementation. In the example in FIG. 1 , radiotherapy treatmentgenerally includes various stages, such as an imaging system performingimage data acquisition for a patient (see 110); a radiotherapy treatmentplanning system (see 130) generating a suitable treatment plan (see 156)for the patient; and a treatment delivery system (see 160) deliveringtreatment according to the treatment plan.

In more detail, at 110 in FIG. 1 , image data acquisition may beperformed using an imaging system to capture image data 120 associatedwith a patient (particularly the patient's anatomy). Any suitablemedical image modality or modalities may be used, such as computedtomography (CT), cone beam computed tomography (CBCT), positron emissiontomography (PET), magnetic resonance imaging (MRI), single photonemission computed tomography (SPECT), any combination thereof, etc. Forexample, when CT or MRI is used, image data 120 may include a series oftwo-dimensional (2D) images or slices, each representing across-sectional view of the patient's anatomy, or may include volumetricor three-dimensional (3D) images of the patient, or may include a timeseries of 2D or 3D images of the patient (e.g., four-dimensional (4D) CTor 4D CBCT).

At 130 in FIG. 1 , radiotherapy treatment planning may be performedduring a planning phase to generate treatment plan 156 based on imagedata 120. Any suitable number of treatment planning tasks or steps maybe performed, such as segmentation, dose prediction, projection dataprediction, treatment plan generation, etc. For example, segmentationmay be performed to generate structure data 140 identifying varioussegments or structures may from image data 120. In practice, athree-dimensional (3D) volume of the patient's anatomy may bereconstructed from image data 120. The 3D volume that will be subjectedto radiation is known as a treatment or irradiated volume that may bedivided into multiple smaller volume-pixels (voxels) 142. Each voxel 142represents a 3D element associated with location (i, j, k) within thetreatment volume. Structure data 140 may be include any suitable datarelating to the contour, shape, size and location of patient's anatomy144, target 146 and any organ-at-risk (OAR) 148.

In another example, dose prediction may be performed to generate dosedata 150 specifying radiation dose to be delivered to target 146(denoted “D_(TAR)” at 152) and radiation dose for OAR 148 (denoted“D_(OAR)” at 154). In practice, target 146 may represent a malignanttumor (e.g., prostate tumor, etc.) requiring radiotherapy treatment, andOAR 148 a proximal healthy structure or non-target structure (e.g.,rectum, bladder, etc.) that might be adversely affected by thetreatment. Target 146 is also known as a planning target volume (PTV).Although an example is shown in FIG. 1 , the treatment volume mayinclude multiple targets 146 and OARs 148 with complex shapes and sizes.Further, although shown as having a regular shape (e.g., cube), voxel142 may have any suitable shape (e.g., non-regular). Depending on thedesired implementation, radiotherapy treatment planning at block 130 maybe performed based on any additional and/or alternative data, such asprescription, disease staging, biologic or radiomic data, genetic data,assay data, biopsy data, past treatment or medical history, anycombination thereof, etc.

Based on structure data 140 and dose data 150, treatment plan 156 may begenerated include 2D fluence map data for a set of beam orientations orangles. Each fluence map specifies the intensity and shape (e.g., asdetermined by a multileaf collimator (MLC)) of a radiation beam emittedfrom a radiation source at a particular beam orientation and at aparticular time. For example, in practice, intensity modulatedradiotherapy treatment (IMRT) or any other treatment technique(s) mayinvolve varying the shape and intensity of the radiation beam while at aconstant gantry and couch angle. Alternatively or additionally,treatment plan 156 may include machine control point data (e.g., jaw andleaf positions), volumetric modulated arc therapy (VMAT) trajectory datafor controlling a treatment delivery system, etc. In practice, block 130may be performed based on goal doses prescribed by a clinician (e.g.,oncologist, dosimetrist, planner, etc.), such as based on theclinician's experience, the type and extent of the tumor, patientgeometry and condition, etc.

At 160 in FIG. 1 , treatment delivery is performed during a treatmentphase to deliver radiation to the patient according to treatment plan156. For example, radiotherapy treatment delivery system 160 may includerotatable gantry 164 to which radiation source 166 is attached. Duringtreatment delivery, gantry 164 is rotated around patient 170 supportedon structure 172 (e.g., table) to emit radiation beam 168 at variousbeam orientations according to treatment plan 156. Controller 162 may beused to retrieve treatment plan 156 and control gantry 164, radiationsource 166 and radiation beam 168 to deliver radiotherapy treatmentaccording to treatment plan 156.

It should be understood that any suitable radiotherapy treatmentdelivery system(s) may be used, such as mechanic-arm-based systems,tomotherapy type systems, brachy, sirex spheres, any combinationthereof, etc. Additionally, examples of the present disclosure may beapplicable to particle delivery systems (e.g., proton, carbon ion,etc.). Such systems may employ either a scattered particle beam that isthen shaped by a device akin to an MLC, or may instead employ a scanningbeam of adjustable energy, spot size, and dwell time.

Conventionally, radiotherapy treatment planning at block 130 in FIG. 1is time and labor intensive. For example, it usually requires a team ofhighly skilled and trained oncologists and dosimetrists to manuallydelineate structures of interest by drawing contours or segmentations onimage data 120. These structures are manually reviewed by a physician,possibly requiring adjustment or re-drawing. In many cases, thesegmentation of critical organs can be the most time-consuming part ofradiation treatment planning. After the structures are agreed upon,there are additional labor-intensive steps to process the structures togenerate a clinically-optimal treatment plan specifying treatmentdelivery data such as beam orientations and trajectories, as well ascorresponding 2D fluence maps. These steps are often complicated by alack of consensus among different physicians and/or clinical sites as towhat constitutes “good” contours or segmentation. In practice, theremight be a huge variation in the way structures or segments are drawn bydifferent clinical experts. The variation may result in uncertainty intarget volume size and shape, as well as the exact proximity, size andshape of OARs that should receive minimal radiation dose. Even for aparticular clinical expert, there might be variation in the way segmentsare drawn on different days.

According to examples of the present disclosure, artificial intelligence(AI) techniques may be applied to ameliorate various challengesassociated with radiotherapy treatment planning. In particular, deeplearning engine(s) may be used to automate radiotherapy treatmentplanning step(s) and/or adaptive radiotherapy treatment planningstep(s). Examples of the present disclosure may be implemented toimprove the efficiency of radiotherapy treatment planning and possiblythe treatment outcome, such as increasing the tumor control probabilityand/or reducing the likelihood of health complications or death due toradiation overdose in the healthy structures. For example, automaticsegmentation of image data 120 would be of great benefit in speeding upthe workflow and enabling various applications, such automatic treatmentplanning and radiotherapy treatment adaptation.

Throughout the present disclosure, the term “deep learning” may refergenerally to a class of approaches that utilizes many layers or stagesof nonlinear data processing for feature learning as well as patternanalysis and/or classification. Accordingly, the term “deep learningmodel” may refer to a hierarchy of layers of nonlinear data processingthat include an input layer, an output layer, and multiple (i.e., two ormore) “hidden” layers between the input and output layers. These layersmay be trained from end-to-end (e.g., from the input layer to the outputlayer) to extract feature(s) from an input and classify the feature(s)to produce an output (e.g., classification label or class).

Accordingly, the term “deep learning engine” may refer to any suitablehardware and/or software component(s) of a computer system that arecapable of executing algorithms according to any suitable deep learningmodel(s). Depending on the desired implementation, any suitable deeplearning model(s) may be used, such as convolutional neural network,recurrent neural network, deep belief network, or any combinationthereof, etc. In practice, a neural network is generally formed using anetwork of processing elements (called “neurons,” “nodes,” etc.) thatare interconnected via connections (called “synapses,” “weights,” etc.).

Deep learning approaches should be contrasted against machine learningapproaches that have been applied to, for example, automaticsegmentation. In general, these approaches involve extracting(hand-designed) feature vectors from images, such as for every voxel,etc. Then, the feature vectors may be used as input to a machinelearning model that classifies which class each voxel belongs to.However, such machine learning approaches usually do not make use ofcomplete image data and additional constraints may be required. Anotherchallenge is that these approaches rely on a high dimension ofhand-designed features in order to accurately predict the class labelfor each voxel. Solving a high-dimensional classification problem iscomputationally expensive and requires a large amount of memory. Someapproaches use lower dimensional features (e.g., using dimensionalityreduction techniques) but they may decrease the prediction accuracy.

In the following, various examples will be discussed below using FIG. 1to FIG. 10 . In particular, radiotherapy treatment planning using deeplearning engine(s) will be discussed using FIG. 1 to FIG. 5 . Further,adaptive radiotherapy treatment planning using deep learning engine(s)will be discussed using FIG. 6 to FIG. 9 . Examples of the presentdisclosure may be implemented using any suitable computer system(s), anexample of which is shown in FIG. 10 .

Deep Learning Engine with Multiple Processing Pathways

According to a first aspect of the present disclosure, radiotherapytreatment planning may be improved using a deep learning engine withmultiple (K) processing pathways to process medical image data atdifferent resolution levels. Some examples will be explained using FIG.2 and FIG. 3 . In particular, FIG. 2 is a schematic diagram illustratingexample deep learning engine 200 with multiple processing pathways forradiotherapy treatment planning. FIG. 3 is a flowchart of exampleprocess 300 for a computer system to perform radiotherapy treatmentplanning using deep learning engine 200. Example process 300 may includeone or more operations, functions, or actions illustrated by one or moreblocks, such as 310 to 370. The various blocks may be combined intofewer blocks, divided into additional blocks, and/or eliminated basedupon the desired implementation.

In the example in FIG. 2 , multi-resolution deep learning engine 200includes at least three (K=3) processing pathways 221-223 to processrespective image data 211-213 at different resolution levels (denoted asR_(k)=R₁, R₂, R₃) to generate output data 260. Image data 211-213 ofmultiple resolution levels may be fed separately intospecifically-tailored processing pathways 221-223. This way, deeplearning engine 200 may achieve a larger receptive field to improve ofprediction outcome compared to conventional approaches that rely on asingle processing pathway. In particular, to achieve the same (i.e.,larger) receptive field, a single-pathway deep learning engine wouldhave to be deeper (e.g., more layers) and therefore require morecomputing power.

In practice, a larger receptive field is better than a smaller one tofacilitate extraction and analysis of both local and global feature datain image data 211-213 to produce better quality output data. In general,deep neural networks may be difficult to tune to work properly formedical image data, as the needed accuracy and reliability is relativelyhigh. By breaking the image processing problem into multiple resolutionlevels, examples of the present disclosure may be implemented in aresource-efficient manner. At a user's site with limited processingresources, for example, memory-efficient approaches are preferred toimprove efficiency. For example, the processing cost is lower at a lowerresolution in the sense that a processing pathway may process moredistant data (e.g., feature data at different physical distances) at thesame cost compared to the case where there is no downsampling.

Referring also to FIG. 3 , at 310, first image data 120/211 associatedwith a patient is obtained. Here, the term “obtain” may refer generallyto a computer system accessing or retrieving image data 120/211 from anysuitable source (e.g., another computer system), memory or storage(e.g., local or remote), etc. In practice, first image data 211 may be2D or 3D image data acquired using any suitable imaging modality ormodalities.

At 320 in FIG. 3 , first image data (I₁) 211 associated with a firstresolution level (R₁) is processed using first processing pathway 221 togenerate first feature data (F₁) 241. At 330 in FIG. 3 , second imagedata (I₂) 212 associated with a second resolution level (R₂) isprocessed using second processing pathway 222 to generate second featuredata (F₂) 242. At 340 in FIG. 3 , third image data (I₃) 213 associatedwith a third resolution level (R₃) is processed using third processingpathway 223 to generate third feature data (F₃) 243. In practice, a“processing pathway” may be implemented using any suitable architecture,such as convolutional block(s) or layer(s) to generate feature data.

At 350 in FIG. 3 , first combined set of feature data (C₁) 251associated with the second resolution level (R₂) is generated based onsecond feature data (F₂) 242 and third feature data (F₃) 243. At 360 inFIG. 3 , second combined set of feature data (C₂) 252 associated withthe first resolution level (R₁) is generated based on first combined set(C₁) 251 and first feature data (F₁) 241. At 370 in FIG. 3 , output data260 associated with radiotherapy treatment of the patient is generatedbased on second combined set (C₂) 252. Examples of the presentdisclosure may be implemented to facilitate better integration offeature data 241-243 of different resolution levels from respectiveprocessing pathways 221-223.

Depending on the desired implementation, deep learning engine 200 may betrained to perform automatic segmentation to generate outputdata=structure data (e.g., 140 in FIG. 1 ), dose prediction to generateoutput=dose data (e.g., 150 in FIG. 1 ), treatment delivery dataestimation to generate output=treatment delivery data, or anycombination thereof. For example, in the case of automatic segmentation,the structure data may include segmented and labeled images specifyingthe contour, shape, size and/or location of structure(s) or segment(s)that are identifiable from image data 120/211, such as patient's anatomy144, target 146, and OAR 148 in FIG. 1 . In the case of dose prediction,the dose data may specify radiation dose for target 146 and OAR 148 (seealso 152-154 in FIG. 1 ). In the case of treatment delivery dataestimation, the “treatment delivery data” may be structure projectiondata (e.g., beam trajectories and/or orientations), fluence map data,etc.

In the example in FIG. 2 , second image data (I₂) 212 may be generatedbased on first image data (I₁) 211, and third image data (I₃) 213 basedon second image data (I₂) 212 or first image data (I₁) 211. Deeplearning engine 200 includes matching resampling (i.e., downsampling orupsampling) blocks 231-234. For example, in the case of R₁>R₂>R₃ (to bediscussed using FIG. 4 and FIG. 5 ), downsampling blocks 231-232 may beused to reduce the resolution level, and matching upsampling blocks233-234 to increase the resolution level.

In the following, examples relating to automatic segmentation will bedescribed using FIG. 4 and dose prediction using FIG. 5 . Althoughexemplified using deep convolutional neural networks, it should beunderstood that any alternative and/or additional deep learning model(s)may be used to implement deep learning engine 200/400/500 according toexamples of the present disclosure.

Automatic Segmentation

FIG. 4 is a schematic diagram illustrating example deep learning engine400 to perform automatic segmentation of image data for radiotherapytreatment planning. Similar to the example in FIG. 2 , deep learningengine 400 includes K=3 processing pathways (see 421-423) that aretrained during training phase 401. Once trained, deep learning engine400 may be used (e.g., by a clinician) to perform automatic segmentationfor actual patients during inference phase 402.

(a) Training Data

During training phase 401, deep learning engine 400 may be trained usingany suitable training data 411-412 relating to automatic segmentation.In practice, training data 411-412 may include example inputdata=unsegmented image data 411, and example output data=structure data412 (also known as segmentation data). Structure data 412 may identifyany suitable contour, shape, size and/or location of structure(s) orsegment(s) of a patient's anatomy, such as target(s), OAR(s), etc. Imagedata 411 may include 2D or 3D images of the patient's anatomy, andcaptured using any suitable imaging modality or modalities. Depending onthe desired implementation, structure data 412 may be manually generatedand clinically validated by trained professionals using any suitableapproach.

The aim of training phase 401 is to train deep learning engine 400 toperform automatic segmentation by mapping input data=image data 411 toexample output data=structure data 412. Training phase 401 may involvefinding weights that minimize the training error between trainingstructure data 412, and estimated structure data 482 generated by deeplearning engine 400. In practice, deep learning engine 200 may betrained identify multiple targets and OARs of any suitable shapes andsizes.

For example, in relation to prostate cancer, image data 411 may includeimage data of a patient's prostate. In this case, structure data 412 mayidentify a target representing the patient's prostate, and an OARrepresenting a proximal healthy structure such as rectum or bladder. Inrelation to lung cancer treatment, image data 411 may include image dataof a patient's lung. In this case, structure data 412 may identify atarget representing cancerous lung tissue, and an OAR representingproximal healthy lung tissue, esophagus, heart, etc. In relation tobrain cancer, image data 411 may include image data of a patient'sbrain, in which case structure data 412 may identify a targetrepresenting a brain tumor, and an OAR representing a proximal opticnerve, brain stem, etc.

In practice, training data 411-412 may be user-generated throughobservations and experience to facilitate supervised learning. Forexample, training data 411-412 may be extracted from past treatmentplans developed for past patients. Training data 411-412 may bepre-processed using any suitable data augmentation approach (e.g.,rotation, flipping, translation, scaling, noise addition, cropping, anycombination thereof, etc.) to produce a new dataset with modifiedproperties to improve model generalization using ground truth. Inpractice, a 3D volume of the patient that will be subjected to radiationis known as a treatment volume, which may be divided into multiplesmaller volume-pixels (voxels). In this case, structure data 412 mayspecify a class label (e.g., “target,” “OAR,” etc.) associated with eachvoxel in the 3D volume. Depending on the desired implementation,structure data 412 may identify multiple targets and OARs of anysuitable shapes and sizes.

(b) Processing Pathways and Layers

Deep learning engine 400 includes three processing pathways 421-423(k=1, 2, 3) to process image data at different resolution levels(R_(k)=R₁, R₂, R₃). First processing pathway 421 (k=1) is configured toprocess first image data (I₁) at a first resolution level R₁ (e.g., 1×).Second processing pathway 422 (k=2) is configured to process secondimage data (I₂) at a second resolution level R₂<R₁ to enlarge thereceptive field. Third processing pathway 423 (k=3) is configured toprocess third image data (I₃) at a third resolution level R₃<R₂<R₁ tofurther enlarge the receptive field.

In the example in FIG. 4 , the input to first processing pathway 421 isimage data 411. The input to second processing pathway 422 is image data411 that has been downsampled (e.g., by a factor of 2×) by downsamplingblock 431. The input to the third processing pathway 423 is image data411 that has been downsampled by both downsampling blocks 431-432 (e.g.,by a total factor of 4×). Downsampling blocks 431-432 have matchingupsampling blocks 441-443 for upsampling before feature data (see F₁,F₂, F₃) from respective processing pathways 421-423 are combined. Inother words, each downsampling step has a corresponding (i.e., matching)upsampling step to readjust the resolution level. Downsampling blocks431-432 may be implemented using subsampling, pooling, etc., andupsampling blocks 441-443 using transposed convolutions, repeating, etc.

By processing image data 411 at multiple resolution levels, processingpathways 421-423 provide different views into image data 411 to achievea larger receptive field. In practice, medical image data generallyincludes both local and global feature data of a patient's anatomy,where the terms “local” and “global” are relative in nature. Forexample, the local feature data may provide a microscopic view of thepatient's anatomy, such as tissue texture, whether a structure has alimiting border, etc. In contrast, the global feature data may provide arelatively macroscopic view of the patient's anatomy, such as whichregion the anatomy is located (e.g., pelvis, abdomen, head and neck,etc.), orientation (e.g., to the left, to the right, front, back), etc.

In the example in FIG. 4 , first processing pathway 421 may processimage data 411 at the highest resolution level (R₁) to analyze localtissue texture. Second processing pathway 422 may process image data 411at an intermediate resolution level (R₂<R₁) to analyze tissue typechanges for evidence of nearby structural boundaries. Third processingpathway 422 may process image data 411 at the coarsest resolution level(R₃<R₂<R₁) to analyze landmarks such as bones and body outline.Processing image data 411 at a lower resolution level generally requiresless processing. This is especially significant for 3D image dataprocessing, where halving the resolution may cut the processing cost to⅛. This allows more resources to be devoted to more accuratesegmentation, such as more channels in processing pathways 421-423.

Using deep convolutional neural networks for example, processingpathways 421-423 may each include any suitable number of convolutionlayers (e.g., 424-426) to extract feature data (F₁, F₂, F₃) at differentresolution levels from image data 411. In practice, each convolutionlayer may be configured to extract feature data (e.g., 2D or 3D featuremap) at a particular resolution level by applying filter(s) or kernel(s)to overlapping regions of its input. Numerical values of parameters inthe convolution filters are learned during training phase 401. Forexample, the convolution layer may create a 2D feature map that includesfeatures that appear in 2D image data, or a 3D feature map for 3D imagedata. This automatic feature extraction approach should be distinguishedfrom conventional approaches that rely on hand-designed features.

Deep learning engine 400 further includes additional convolution layersor blocks 450-470 and mixing blocks 480 (one shown for simplicity) tocombine feature data (F₁, F₂, F₃) from processing pathways 421-423 in astaged manner. In particular, third feature data (F₃) from thirdprocessing pathway 423 may be upsampled from the lowest resolution levelR₃ to the intermediate resolution level R₂ using upsampling block 441.The upsampled third feature data (F₃) is then combined with secondfeature data (F₂) from second processing pathway 422 using convolutionalblock 450, thereby generating first combined set (C₁). As anoptimization strategy, convolutional block 450 may be configured to“smooth” or “refine” the second feature data (F₂) and upsampled thirdfeature data (F₃) before another stage of upsampling (e.g., 2×) isperformed using subsequent upsampling blocks 442-443.

The feature data (F₁, F₂, F₃) from all processing pathways 421-423 arethen combined using additional convolutional blocks 460-470, therebygenerating second combined set (C₂). In particular, the feature data maybe combined by upsampling a lower resolution path to the resolution of ahigher resolution path. To bring different feature data to the sameresolution level, upsampling blocks 442-443 may be used to upsamplefirst combined set (C₁) from convolutional block 450. In practice,convolutional blocks included in processing pathways 421-423, as well asconvolutional blocks 450-470 may be of any suitable configuration (e.g.,3×3×3 convolutions).

Second combined set (C₂) generated using convolutional blocks 460-470 isthen processed using mixing block 480 to produce output data=estimatedstructure data 482. Mixing block(s) 480 is configured to massage (e.g.,via 1×1×1 convolutions) the final set of features into the finalsegmentation decision (i.e., estimated structure data 482). Estimatedstructure data 482 may specify such as voxel-based classification dataassociated with a treatment volume identified from image data 411. Forexample, a voxel may be classified as a target (e.g., label=“TAR”) or anOAR (e.g., label=“OAR”). In practice, label=“OAR” may represent a largergroup of labels, such as “Rectum,” “Bladder,” “Brainstem,” or any otheranatomically-defined volume. Further, label=“TAR” may represent a tumoror treatment volume.

The above training steps may be repeated during training phase 401 tominimize the error between the expected result in training structuredata 412 and estimated structure data 482. Depending on the desiredimplementation, deep learning engine 400 may be implemented using anysuitable convolutional neural network architecture(s), such as U-net,LeNet, AlexNet, ResNet, V-net, DenseNet, etc. For example, the U-netarchitecture includes a contracting path (left side) and an expansivepath (right side). The contracting path includes repeated application ofconvolutions, followed by a rectified linear unit (ReLU) and max pollingoperation(s). Each step in the expansive path may include upsampling ofthe feature map followed by convolutions, etc. It should be noted thatprocessing pathways 421-423 may use the same architecture, or differentones.

(c) Inference Phase

Once trained, deep learning engine 400 may be used by a clinician duringinference phase 402 to perform segmentation to generate outputdata=patient structure data 260/492 based on input data=image data210/491 of a particular patient. Image data 210/491 may be processed byprocessing pathways 421-423 of deep learning engine 400 at respectiveresolution levels to enlarge the receptive field. The example process(see blocks 310-370) explained using FIG. 3 may be applied to performautomatic segmentation, and will not be repeated here for brevity.

Dose Prediction

FIG. 5 is a schematic diagram illustrating example deep learning engine500 to perform dose prediction for radiotherapy treatment planning.Similar to the example in FIG. 4 , deep learning engine 500 includes K=3processing pathways (see 521-523) that are trained during training phase501. Once trained, deep learning engine 500 may be used (e.g., by aclinician) to perform dose prediction for actual patients duringinference phase 502.

(a) Training Data

During training phase 501, deep learning engine 500 may be trained usingany suitable training data 511-512 relating to dose prediction. Inpractice, training data 511-512 may include example input data=imagedata and structure data 511 (i.e., segmented image data), and exampleoutput data=dose data 512. Dose data 512 (e.g., 3D dose data) mayspecify dose distributions for a target (denoted “D_(TAR)”) and an OAR(denoted “D_(OAR)”). In practice (not shown in FIG. 5 for simplicity),dose data 512 may specify the dose distributions for the whole 3Dvolume, not just the target and OAR volumes. Depending on the desiredimplementation, dose data 512 may include spatial biological effect data(e.g., fractionation corrected dose) and/or cover only part of thetreatment volume.

For example, in relation to prostate cancer, dose data 512 may specifydose distributions for a target representing the patient's prostate, andan OAR representing a proximal healthy structure such as rectum orbladder. In relation to lung cancer treatment, dose data 512 may specifydose distributions for a target representing cancerous lung tissue, andan OAR representing proximal healthy lung tissue, esophagus, heart, etc.In relation to brain cancer, dose data 512 may specify dosedistributions for a target representing a brain tumor, and an OARrepresenting a proximal optic nerve, brain stem, etc.

The aim of training phase 501 is to train deep learning engine 500 toperform dose prediction by mapping input data=image data andcorresponding structure data 511 to example output data=dose data 512.Training phase 501 may involve finding weights (e.g., kernel parameters)that minimize the training error between training dose data 512, andestimated dose data 582 generated by deep learning engine 500. Anysuitable constraint(s) may be used, such as limiting dose prediction tothe vicinity of target(s) or certain dose levels only.

(b) Processing Pathways and Layers

Similar to the example in FIG. 4 , deep learning engine 500 in FIG. 5includes three processing pathways 521-523 (k=1,2,3) to process imagedata at different resolution levels (R_(k)=R₁, R₂, R₃). First processingpathway 521 (k=1) is configured to process first image data (I₁) at afirst resolution level R₁ (e.g., 1×). Second processing pathway 522(k=2) is configured to process second image data (I₂) at a secondresolution level R₂<R₁ to enlarge the receptive field. Third processingpathway 523 (k=3) is configured to process third image data (I₃) at athird resolution level R₃<R₂<R₁ to further enlarge the receptive field.

In the example in FIG. 5 , the input to first processing pathway 521 isimage data 511. The input to second processing pathway 522 is image data(I₂) that has been downsampled (e.g., by a factor of 2×) by downsamplingblock 531. The input to the third processing pathway 523 is image data(I₃) that has been downsampled by both downsampling blocks 531-532(e.g., by a total factor of 6×). Downsampling blocks 531-532 havematching upsampling blocks 541-543 for upsampling before feature data(see F₁, F₂, F₃) from respective processing pathways 521-523 arecombined.

Deep learning engine 500 further includes additional convolution layersor blocks 550-570 and mixing blocks 580 (one shown for simplicity) tocombine feature data (F₁, F₂, F₃) from processing pathways 521-523 instages. Similarly, third feature data (F₃) may be upsampled usingupsampling block 541 (e.g., by a factor of 4×) before being combinedwith second feature data (F₂) using convolutional block 550, therebygenerating first combined set (C₁). Further, first combined set (C₁) maybe upsampled using upsampling blocks 542-543 (e.g., by a factor of 2×)before being combined with first feature data (F₁) using convolutionalblocks 560-570, thereby generating second combined set (C₂). Mixingblock(s) 580 is configured to massage (e.g., using 1×1×1 convolutions)the final set of features into the final dose prediction decision (i.e.,estimated dose data 582).

(c) Inference Phase

Once trained, deep learning engine 500 may be used by a clinician duringinference phase 502 to perform dose prediction to generate outputdata=dose data 260/592 based on input data=image data 210/591 of aparticular patient. Image data 210/591 may be processed by processingpathways 521-523 of deep learning engine 500 at respective resolutionlevels to enlarge the receptive field. The example process (see blocks310-370) explained using FIG. 3 may be applied to perform doseprediction, and will not be repeated here for brevity.

(d) Variations

In practice, deep learning engine 200/400/500 may be trained to processdata relating to any suitable number of resolution levels. In practice,the number of processing pathways and corresponding resolution levelsmay depend on the input image data. For example, at some point,downsampling may not reveal additional features of interest because thedata would be too coarse. Medical image data resolution tends to bequite high, and three or more resolution levels may be appropriate toachieve efficiency gains.

In the case of K=4, a fourth processing pathway may be used to processfourth image data (I₄) associated with a fourth resolution level. Forexample, the fourth image data (I₄) may be generated by downsampling thefirst image data (I₁), second image data (I₂) or third image data (I₃)using any suitable downsampling factor. Feature data (F₁, F₂, F₃, F₄)from respective K=4 processing pathways may be combined in staged mannerto improve efficiency. For example, F₄ and F₃ may be combined first,followed by F₂, and finally F₁ (e.g., in the order of F_(K), . . . ,F₁).

Besides automatic segmentation in FIG. 4 and dose prediction in FIG. 5 ,examples of the present disclosure may be implemented to performtreatment delivery data prediction. In this case, the treatment deliverydata (i.e., output data) may include structure projection data, fluencemap data, etc. For example, deep learning engine 200 may be trained toperform structure projection data, such as based on image data,structure data, dose data, or any combination thereof. The structureprojection data may include data relating to beam orientations andmachine trajectories for a treatment delivery system (see 160 in FIG. 1). In another example, deep learning engine 200 may be trained toperform fluence map estimation, such as 2D fluence maps for a set ofbeam orientations/trajectories, machine control point data (e.g., jawand leaf positions, gantry and couch positions), etc. Fluence maps willbe explained further using FIG. 9 .

Input data and output data of deep learning engine 200/400/500 mayinclude any suitable additional and/or alternative data. For example,field geometry data could be input or outputs for all applications.Other examples include monitor units (amount of radiation counted bymachine), quality of plan estimate (acceptable or not), daily doseprescription (output), field size or other machine parameters, couchpositions parameters or isocenter position within patient, treatmentstrategy (use movement control mechanism or not, boost or no boost),treat or no treat decision.

Adaptive Radiotherapy (ART)

In radiotherapy, the treatment goal is to be able to deliver a high doseto the target (e.g., to kill cancer cells) while sparing the healthytissue (e.g., to minimize adverse effect on critical OARs). As such, itis important to deliver to the correct spot during the span of theradiotherapy treatment. However, the situation or condition of apatient's anatomy at the time of delivery might differ considerably fromthat considered in a treatment plan. For example, the shape, size andposition of critical organs might have changed compared to those in theplanning image data (e.g., CT images). The difference might be caused byvarious factors, such as internal organ movement (e.g., bladder filing,bowel movement), patient's weight loss, tumor shrinkage or expansion,etc. In certain cases, the existing treatment plan that is generatedbased on the planning image data may no longer satisfy the goal of thetreatment, and a new treatment plan is required. This is known as ART.

For example, CT image data is usually acquired during a planning phase(i.e., prior to a treatment phase) for the purpose of treatmentplanning. A treatment plan may be generated based on manual segmentationof the CT image data. During the treatment phase (e.g., near or at thetime of treatment delivery), CBCT image data may be acquired to monitorany changes in the patient's condition. A clinician may compare the CBCTimage data with the CT image data to assess whether the treatment planis still applicable to produce precise dose delivery. If the treatmentplan is no longer satisfying the treatment goal, the treatment planneeds to be adjusted.

Conventionally, ART generally involves the clinician repeating themanual segmentation step on the newly acquired CBCT image data toimprove the quality of the treatment plan. Depending on the case and/ortreatment area, segmentation is easily one of the costliest bottlenecksin ART because the number of structures and the complexity of theirshapes may vary. For example, contouring may take from few minutes tofew hours. In some cases, the patient may not be treated in a timelymanner because re-scan may be required to continue the planning processoffline. The patient cannot continue the treatment until the new plan isready, which has the undesirable effect of delaying treatment.

According to examples of the present disclosure, ART planning may beimproved using deep learning engines. In the following, two exampleapproaches will be explained. The first approach according to FIG. 6 andFIG. 7 may be implemented when the CBCT image data (“treatment imagedata”) acquired during a treatment phase is significantly different fromthe CT image data (“planning image data”) acquired prior to thetreatment phase. Otherwise, the second approach according to FIG. 6 andFIG. 7 may be implemented.

In more detail, FIG. 6 is a schematic diagram illustrating exampleprocess flow 600 for a computer system to perform adaptive radiotherapytreatment planning using a deep learning engine. Example process 600 mayinclude one or more operations, functions, or actions illustrated by oneor more blocks. The various blocks may be combined into fewer blocks,divided into additional blocks, and/or eliminated based upon the desiredimplementation. Examples of the present disclosure may be implementedusing any suitable computer system, an example of which will bediscussed using FIG. 10 .

At 610 and 620 in FIG. 6 , treatment image data associated with a firstimaging modality (e.g., CBCT image data), and planning image dataassociated with a second imaging modality (e.g., CT image data) may beobtained. Here, the term “obtain” may refer to a computer systemaccessing or retrieving image data from any suitable source (e.g.,another computer system, local memory/storage, remote memory/storage,etc.). The term “treatment image data” may refer generally to anysuitable image data that may be acquired during treatment phase 601(e.g., close to, or on, the day of a scheduled treatment) to determinewhether ART is required. The term “planning image data” may refergenerally to any suitable image data that may be acquired duringplanning phase 602 (i.e., prior to the treatment phase 601) for thepurpose of generating a treatment plan (see 603) for the patient.

Next, treatment image data 610 and planning image data 620 may becompared to determine whether an update of the treatment plan generatedbased on the planning image data is required. If yes (i.e., updaterequired), either a first approach (see 640-660) or a second approach(see 670-690) may be implemented based on whether their differenceexceeds a significance threshold. In particular, at 630 in FIG. 3 , thefirst approach may be implemented in response to determination that adifference between treatment image data 610 and planning image data 620exceeds a predetermined significance threshold. Otherwise, at 632 inFIG. 6 , in response to determination that their difference does notexceed the predetermined significance threshold, the second approach maybe implemented. If the patient's condition has changed significantlysince the planning phase 602, the first approach may be implementedbased on treatment image data 610. If the difference is lesssignificant, the second approach may be implemented to take advantage ofthe information in both treatment image data 610 and planning image data620.

The selection between the first approach and the second approach may beperformed manually (e.g., by a clinician) or programmatically (e.g., bya computer system). The “predetermined significance threshold” may beassociated with (e.g., set based on, relating to) at least one of thefollowing: shape, size or position change of a target requiring dosedelivery; and shape, size or position change of healthy tissue (e.g.,OAR) proximal to the target. Depending on the relevant clinicalexpertise, any suitable quality metric data may be used to assessdistance or error mapping between treatment image data 610 and planningimage data 620, such as target size, shift in tumor position (e.g., theposition of voxels associated with the target in 3D mapping), distancefrom target to OARs (e.g., distance to surface or centroid), dosimetricvalues in target and OARs if the original field setup is used in the newsituation, etc.

It should be understood that the examples in FIG. 6 to FIG. 8 areapplicable to image data acquired any suitable imaging modality ormodalities (i.e., not limited to CT and CBCT image data). For example,treatment image data 610 may be in the form of CBCT image data, andplanning image data 620 in the form of CT image data, ultrasound imagedata, MRI image data, PET image data, SPECT or camera image data (e.g.,using a time of flight camera to capture the patient's body outline),etc. In practice, there are generally limited options to acquiretreatment image data 610 given the limited capabilities of the treatmentdelivery machine. For example, CBCT image data acquired during treatmenthas a relatively degraded image quality compared to CT image dataacquired for treatment planning purposes. The area of the patient'sanatomy scanned by a CBCT is generally smaller than the area of the CT,thus some structures might not be fully visible in the CBCT image data.

In the case of treatment image data 610=CT image data (e.g., associatedwith one energy level), planning image data 620 may be in the form of CTimage data associated with a different energy level, ultrasound imagedata, MRI image data, PET image data, SPECT image data or camera imagedata. In the case of treatment image data 610=MRI image data, planningimage data 620 may be in the form of CT image data, CBCT image data,ultrasound image data, MRI image data, PET image data, SPECT image dataor camera image data. In the case of treatment image data 610=ultrasoundimage data, planning image data 620 may be in the form of CT image data,CBCT image data, PET image data, MRI image data, SPECT image data orcamera image data. In the case of treatment image data 610=PET imagedata, planning image data 620 may be in the form of CT image data, CBCTimage data, ultrasound image data, MRI image data, SPECT image data orcamera image data. Alternative and/or additional image data associatedwith any suitable imaging modality or modalities may be used.

Further, it should be understood that deep learning engine 650/680 inthe examples in FIG. 6 to FIG. 8 may be implemented using any suitabledeep learning technique(s). In the following, the deep learningarchitecture with multiple processing pathways in FIG. 1 to FIG. 5 willbe used as an example. Depending on the desired implementation, anyalternative and/or additional deep learning model(s) may be used (i.e.,single or multiple processing pathways). Examples of the presentdisclosure may be implemented to improve the efficiency of ART, whichmay improve customer satisfaction, increase the number of patients thatcan be treated and maintain a set level of planning quality forpatients. In the following, CBCT image data will be used as an example“treatment image data associated with a first imaging modality” and CTimage as example “treatment image data associated with a second imagingmodality.”

(a) First Approach (Difference>Significance Threshold)

In the example in FIG. 6 , the first example approach may be performedin response to determination that an update of a treatment plangenerated based on planning image data 620 is required, and thedifference between treatment image data 610 and planning image data 620exceeds a predetermined significance threshold. Since there is asignificant deviation, it is not necessary to rely on planning imagedata 620 during ART to avoid any adverse effect on the treatmentdelivery.

In particular, at 640 in FIG. 6 , treatment image data 610 (e.g., CBCTimage data) may be transformed to generate transformed image dataassociated with the second imaging modality (e.g., synthetic CT imagedata). At 650 in FIG. 6 , transformed image data 640 may be processedusing a deep learning engine to generate any suitable output data forupdating treatment plan 603. The output data may be patient structuredata (e.g., identifying one or more targets and/or OARs, etc.)associated with the patient, dose data associated with the patient(e.g., dose distributions for one or more targets and/or OARs),treatment delivery data (e.g., beam orientations and/or trajectories,machine control point data, fluence maps, etc.) associated with atreatment delivery system, or any combination thereof.

Example implementation of the first approach according to blocks 640-660in FIG. 6 will be explained using FIG. 7 , which is a schematic diagramillustrating first example approach 700 FIG. 7 for ART planningaccording to the example in FIG. 6 . Example process 700 may include oneor more operations, functions, or actions illustrated by one or moreblocks. The various blocks may be combined into fewer blocks, dividedinto additional blocks, and/or eliminated based upon the desiredimplementation. Example process 700 may be implemented using anysuitable computer system, an example of which will be discussed usingFIG. 10 .

During training phase 701, deep learning engine 650 may be trained togenerate output data 660 using any suitable training data, such astraining CT image data (see 731) and corresponding output data. In thecase of automatic segmentation, training structure data 732 for CT imagedata 731 may be used. Alternatively (not shown in FIG. 7 forsimplicity), deep learning engine 650 may be trained to perform generatestructure data 732, dose prediction to generate dose data, treatmentdelivery data estimation to generate treatment delivery data, or anycombination thereof. Using the examples in FIG. 2 and FIG. 3 , deeplearning engine 650 may include multiple processing pathways to processimage data (I₁, I₂, I₃) at different resolution levels (R₁, R₂, R₃). Theexamples discussed using FIG. 1 to FIG. 5 are also applicable here andwill not be repeated for brevity. Using deep learning engine 650 insteadof manual approaches, the efficiency of ART may be improved.

In the example in FIG. 7 , transformed image data 640 may be generatedusing deep learning engine 720 that is trained to map image dataassociated with one imaging modality (e.g., CBCT) to another imagingmodality (e.g., CT). For example in FIG. 7 , deep learning engine 720may be trained to map CBCT image data to CT image data. In one example,deep learning engine 720 may be trained using training data thatincludes CT image data and corresponding structure data (see 711), aswell as CBCT image data and corresponding structure data (see 712). Theexamples discussed using FIG. 1 to FIG. 5 are also applicable here andwill not be repeated for brevity. Alternatively, algorithmic approachesmay be used instead of deep learning engine 720, such as rigid ordeformable registration algorithms, etc.

During inference phase 702, treatment planning data 610 (e.g., CBCTimage data) may be processed using trained deep learning engine 720 togenerate transformed image data 640 (e.g., synthetic CT image data).Next, transformed image data 640 may be processed using deep learningengine 650 to generate output data 660. For example in FIG. 7 , deeplearning engine 650 may be trained to perform automatic segmentation togenerate output=structure data identifying target(s) and OAR(s)associated with the patient. Alternatively, deep learning engine 650 maybe trained to perform dose prediction, projection data estimation, etc.Output data 660 may then be used to update treatment plan 603 to reflectchanges in the patient's condition, thereby improving treatment deliveryquality. Treatment may then be delivered based on improved treatmentplan 604 in FIG. 6 .

(b) Second Approach (Difference≤Significance Threshold)

Referring to FIG. 6 again, the second example approach may be performedin response to determination that an update of a treatment plangenerated based on planning image data 620 is required, and thedifference between treatment image data 610 and planning image data 620does not exceed a predetermined significance threshold. Since thedifference is less significant, ART benefits from two sets of imagedata, i.e., both treatment image data 610 and planning image data 620.This is because, for example, CT image data may include additional databecause it does not suffer from severe artifacts compared to CBCT imagedata.

In more detail, at 680 and 690 in FIG. 6 , treatment image data 610 andplanning image data 620 may be processed using a deep learning engine togenerate any suitable output data for updating treatment plan 603.Similarly, output data 690 may include patient structure data (e.g.,identifying one or more targets and OARs, etc.) associated with thepatient, dose data associated with the patient (e.g., dose distributionsfor one or more targets and OARs), treatment delivery data (e.g., beamorientations and/or trajectories, machine control point data, fluencemap data, etc.) associated with a treatment delivery system, or anycombination thereof.

Prior to the processing using deep learning engine 680, treatment imagedata 610 may be transformed to generate transformed image data (see670), such as by performing image registration to register treatmentimage data 610 against planning image data 620, etc. Any suitableapproach for image registration may be used, such as algorithmicapproach, machine learning approach, deep learning approach, etc. Imageregistration may be performed to obtain a correspondence betweentreatment image data 610 and planning image data 620.

For example, after CBCT image data has been deformed to match CT imagedata, they may be fed into deep learning engine 680 to generate outputdata 690. In practice, image registration may be performed using anysuitable approach, such as deep learning approach, algorithms, etc. Oneexample deep learning approach for image registration is disclosed in apaper entitled “Quicksilver: Fast Predictive Image Registration—a DeepLearning Approach” (2017) authored by Xiao, Y., Kwitt, R., Styner, M.,Niethammer, M., and published in Neurolmage (vol. 158, 2017, pages378-396). Such approach may be implemented to perform deformable imageregistration using patch-wise prediction of a deformation model based onimage appearance. A deep encoder-decoder network may be used as theprediction model.

It should be noted that transformed image data 670 in the secondapproach is generated based on both treatment image data 610 andplanning image data 620 (i.e., two inputs, such as CT and CBCT imagedata). This should be contrasted against the first approach, in whichtransformed image data 640 is generated based on one input=treatmentimage data 610 (e.g., CBCT image data of the day). Both approaches mayrely on image registration for the transformation.

Example implementation of the second approach according to blocks670-690 in FIG. 6 will be explained using FIG. 8 , which is a schematicdiagram illustrating second example approach 800 FIG. 8 for ART planningaccording to the example in FIG. 6 . Example process 800 may include oneor more operations, functions, or actions illustrated by one or moreblocks. The various blocks may be combined into fewer blocks, dividedinto additional blocks, and/or eliminated based upon the desiredimplementation. Example process 800 may be implemented using anysuitable computer system, an example of which will be discussed usingFIG. 10 .

During training phase 801, deep learning engine 680 may be trained togenerate output data 690 using any suitable training data, such astraining CT image and structure data 811, as well as training CBCT imageand structure data 812. The aim is to train deep learning engine 680 togenerate output data (e.g., structure data in the case of automaticsegmentation) based on two sets of image data acquired using differentimaging modalities, such as CT and CBCT in FIG. 8 .

Deep learning engine 680 may be implemented using any suitable deeplearning model. Using the examples in FIG. 1 to FIG. 5 , deep learningengine 680 may include multiple processing pathways to process both setsof CT and CBCT image data (I₁, I₂, I₃) at different resolution levels(R₁, R₂, R₃). Compared to using a single set of image data, the inputimage data to deep learning engine 680 may have two values representingthe CT and CBCT image data respectively. Convolutional layers in deeplearning engine 680 may be configured to transform the input image datainto more or less abstract features by combining data from allmodalities. Additional implementation details discussed using FIG. 1 toFIG. 5 are also applicable here and will not be repeated for brevity.

During inference phase 802, trained deep learning engine 680 to processtwo sets of image data, i.e., planning image data 620 and transformedimage data 670 generated using image registration, etc. In the case ofautomatic segmentation, output data 690 may include structure dataidentifying target(s) and OAR(s) associated with the patient.Alternatively, deep learning engine 680 may be trained to perform doseprediction, treatment delivery data estimation, etc. Output data 690 maythen be used to update treatment plan 603 to reflect changes in thepatient's condition, thereby improving treatment delivery quality.Treatment may then be delivered based on improved treatment plan 604 inFIG. 6 .

Using multiple sets of image data acquired using different imagingmodalities, improved output data (e.g., better quality contours) may beproduced than having just one set of image data. Compared to the firstapproach, the two different imaging technologies generally provide moreinformation compared to one imaging technology. For example, time offlight camera system provides information about patient surface from alarge area but not information from inside patient, while CBCT providesinformation from inside patient but for a limited field of view, time offlight camera system capturing movement and CBCT. These two sets ofimage data may be interpreted by deep neural network technology toprovide information in one agreed format (for example CT image, CT imageand segmentation, segmentations, 3D density map, 3d density map withmovements, segmentations with movements, etc.).

Example Treatment Plan

FIG. 9 is a schematic diagram of example treatment plan 156/900generated or improved based on output data in the examples in FIG. 1 toFIG. 8 . Treatment plan 156 may be delivered using any suitabletreatment delivery system that includes radiation source 910 to projectradiation beam 920 onto treatment volume 960 representing the patient'sanatomy at various beam angles 930. Although not shown in FIG. 9 forsimplicity, radiation source 910 may include a linear accelerator toaccelerate radiation beam 920 and a collimator (e.g., MLC) to modify ormodulate radiation beam 920. In another example, radiation beam 920 maybe modulated by scanning it across a target patient in a specificpattern with various energies and dwell times (e.g., as in protontherapy). A controller (e.g., computer system) may be used to controlthe operation of radiation source 920 according to treatment plan 156.

During treatment delivery, radiation source 910 may be rotatable using agantry around a patient, or the patient may be rotated (as in someproton radiotherapy solutions) to emit radiation beam 920 at variousbeam orientations or angles relative to the patient. For example, fiveequally-spaced beam angles 930A-E (also labelled “A,” “B,” “C,” “D” and“E”) may be selected using a deep learning engine configured to performtreatment delivery data estimation. In practice, any suitable number ofbeam and/or table or chair angles 930 (e.g., five, seven, etc.) may beselected. At each beam angle, radiation beam 920 is associated withfluence plane 940 (also known as an intersection plane) situated outsidethe patient envelope along a beam axis extending from radiation source910 to treatment volume 960. As shown in FIG. 9 , fluence plane 940 isgenerally at a known distance from the isocenter.

During radiotherapy treatment planning, treatment plan 156/900 may begenerated based on output data 260/492/592 generated using deep learningengine 200/400/500 in the examples in FIG. 1 to FIG. 5 . During ARTplanning, treatment plan 156/900 may be improved based on output data660/690 generated using deep learning engine 650/680 in the examples inFIG. 6 to FIG. 8 .

Computer System

The above examples can be implemented by hardware, software or firmwareor a combination thereof. FIG. 10 is a schematic diagram of examplecomputer system 1000 for radiotherapy treatment planning and/or ARTplanning. In this example, computer system 1005 (also known as atreatment planning system) may include processor 1010, computer-readablestorage medium 1020, interface 1040 to interface with radiotherapytreatment delivery system 160, and bus 1030 that facilitatescommunication among these illustrated components and other components.

Processor 1010 is to perform processes described herein with referenceto FIG. 1 to FIG. 9 . Computer-readable storage medium 1020 may storeany suitable information 1022, such as information relating to trainingdata, deep learning engines, image data, output data, etc.Computer-readable storage medium 1020 may further storecomputer-readable instructions 1024 which, in response to execution byprocessor 1010, cause processor 1010 to perform processes describedherein. Treatment may be delivered according to treatment plan 156 usingtreatment planning system 160 explained using FIG. 1 , the descriptionof which will not be repeated here for brevity.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. Throughout the presentdisclosure, the terms “first,” “second,” “third,” etc. do not denote anyorder of importance, but are rather used to distinguish one element fromanother.

Those skilled in the art will recognize that some aspects of theembodiments disclosed herein, in whole or in part, can be equivalentlyimplemented in integrated circuits, as one or more computer programsrunning on one or more computers (e.g., as one or more programs runningon one or more computer systems), as one or more programs running on oneor more processors (e.g., as one or more programs running on one or moremicroprocessors), as firmware, or as virtually any combination thereof,and that designing the circuitry and/or writing the code for thesoftware and or firmware would be well within the skill of one of skillin the art in light of this disclosure.

Although the present disclosure has been described with reference tospecific exemplary embodiments, it will be recognized that thedisclosure is not limited to the embodiments described, but can bepracticed with modification and alteration within the spirit and scopeof the appended claims. Accordingly, the specification and drawings areto be regarded in an illustrative sense rather than a restrictive sense.

1. A method for a computer system to perform adaptive radiotherapytreatment planning, wherein the method comprises: obtaining treatmentimage data associated with a first imaging modality, wherein thetreatment image data is acquired during a treatment phase of a patient;obtaining planning image data associated with a second imaging modality,wherein the planning image data is acquired prior to the treatment phaseto generate a treatment plan for the patient; determining whether adifference between the treatment image data and the planning image dataexceeds a threshold; in response to determination that the differenceexceeds the threshold, transforming the treatment image data associatedwith the first imaging modality to generate transformed image dataassociated with the second imaging modality; and processing, using afirst deep learning engine, the transformed image data to generateoutput data for updating the treatment plan; otherwise, processing,using a second deep learning engine, the treatment image data and theplanning image data to generate output data for updating the treatmentplan.
 2. The method of claim 1, wherein determining whether thedifference between the treatment image data and the planning image dataexceeds a threshold comprises: based on a comparison between thetreatment image data and the planning image data, determining whetherthe difference between the treatment image data and the planning imagedata exceeds the threshold in the form of a predetermined significancethreshold relating to at least one of the following: shape, size orposition change of a target requiring dose delivery; and shape, size orposition change of healthy tissue proximal to the target.
 3. The methodof claim 1, wherein transforming the treatment image data comprises:transforming, using a further deep learning engine, the treatment imagedata to generate the transformed image data.
 4. The method of claim 1,wherein transforming the treatment image data comprises: transforming,using an image registration algorithm, the treatment image data to thetransformed image data that includes data suitable for treatmentplanning.
 5. The method of claim 1, wherein processing the transformedimage data using the first deep learning engine comprises: processing,using a first processing pathway of the first deep learning engine, thetransformed image data to generate first feature data associated with afirst resolution level; processing, using a second processing pathway ofthe first deep learning engine, the transformed image data to generatesecond feature data associated with a second resolution level;processing, using a third processing pathway of the first deep learningengine, the transformed image data to generate third feature dataassociated with a third resolution level; and generating the output databased on the first feature data, second feature data and third featuredata.
 6. The method of claim 1, wherein the method further comprises:prior to processing the treatment image data and the planning image datausing the second deep learning engine, performing image registration togenerate transformed image data by registering the treatment image dataagainst the planning image data.
 7. The method of claim 6, whereinprocessing the treatment image data and the planning image data usingthe second deep learning engine comprises: processing, using the seconddeep learning engine, the transformed image data and the planning imagedata to generate the output data.
 8. The method of claim 1, whereinprocessing the treatment image data and the planning image data usingthe second deep learning engine comprises: processing, using a firstprocessing pathway of the second deep learning engine, the treatmentimage data and the planning image data to generate first feature dataassociated with a first resolution level; and processing, using a secondprocessing pathway of the second deep learning engine, the treatmentimage data and the planning image data to generate second feature dataassociated with a second resolution level; processing, using a thirdprocessing pathway of the second deep learning engine, the treatmentimage data and the planning image data to generate third feature dataassociated with a third resolution level; and generating the output databased on the first feature data, second feature data and third featuredata.
 9. The method of claim 1, wherein obtaining the treatment imagedata and the planning image data comprises one of the following:obtaining the treatment image data in the form of cone beam computedtomography (CBCT) image data, and the planning image data in the form ofcomputed tomography (CT) image data, ultrasound image data, magneticresonance imaging (MRI) image data, positron emission tomography (PET)image data, single photon emission computed tomography (SPECT) or cameraimage data; obtaining the treatment image data in the form of CT imagedata, and the planning image data in the form of CT image dataassociated with a different energy level, ultrasound image data, MRIimage data, PET image data, SPECT image data or camera image data;obtaining the treatment image data in the form of MRI image data, andthe planning image data in the form of CT image data, CBCT image data,ultrasound image data, MRI image data, PET image data, SPECT image dataor camera image data; obtaining the treatment image data in the form ofultrasound image data, and the planning image data in the form of CTimage data, CBCT image data, PET image data, MRI image data, SPECT imagedata or camera image data; and obtaining the treatment image data in theform of PET image data, and the planning image data in the form of CTimage data, CBCT image data, ultrasound image data, MRI image data,SPECT image data or camera image data.
 10. The method of claim 1,wherein the method further comprises: training the first or second deeplearning engine to perform one of the following using training dataassociated with past patients: automatic segmentation to generate theoutput data in the form of structure data associated with the patient,dose prediction to generate the output data in the form of dose dataassociated with the patient, and treatment delivery data estimation togenerate the output data in the form of treatment delivery data for atreatment delivery system.
 11. A computer system configured to performadaptive radiotherapy treatment planning, the computer systemcomprising: a processor; and a non-transitory computer-readable mediumhaving stored thereon instructions that, in response to execution by theprocessor, cause the processor to: obtain treatment image dataassociated with a first imaging modality, wherein the treatment imagedata is acquired during a treatment phase of a patient; obtain planningimage data associated with a second imaging modality, wherein theplanning image data is acquired prior to the treatment phase to generatea treatment plan for the patient; determine whether a difference betweenthe treatment image data and the planning image data exceeds athreshold; in response to determination that the difference exceeds thethreshold, transform the treatment image data associated with the firstimaging modality to generate transformed image data associated with thesecond imaging modality; and process, using a first deep learningengine, the transformed image data to generate output data for updatingthe treatment plan; otherwise, process, using a second deep learningengine, the treatment image data and the planning image data to generateoutput data for updating the treatment plan.
 12. The system of claim 11,wherein the instructions for determining whether the difference betweenthe treatment image data and the planning image data exceeds athreshold, in response to execution by the processor, cause theprocessor to: based on a comparison between the treatment image data andthe planning image data, determine whether the difference between thetreatment image data and the planning image data exceeds the thresholdin the form of a predetermined significance threshold relating to atleast one of the following: shape, size or position change of a targetrequiring dose delivery; and shape, size or position change of healthytissue proximal to the target.
 13. The system of claim 11, wherein theinstructions for transforming the treatment image data, in response toexecution by the processor, cause the processor to: transform, using afurther deep learning engine, the treatment image data to generate thetransformed image data.
 14. The system of claim 11, wherein theinstructions for transforming the treatment image data, in response toexecution by the processor, cause the processor to: transform, using animage registration algorithm, the treatment image data to thetransformed image data that includes data suitable for treatmentplanning.
 15. The system of claim 11, wherein the instructions forprocessing the transformed image data using the first deep learningengine, in response to execution by the processor, cause the processorto: process, using a first processing pathway of the first deep learningengine, the transformed image data to generate first feature dataassociated with a first resolution level; process, using a secondprocessing pathway of the first deep learning engine, the transformedimage data to generate second feature data associated with a secondresolution level; process, using a third processing pathway of the firstdeep learning engine, the transformed image data to generate thirdfeature data associated with a third resolution level; and generate theoutput data based on the first feature data, second feature data andthird feature data.
 16. The system of claim 11, wherein thenon-transitory computer-readable medium having stored thereon additionalinstructions that, in response to execution by the processor, cause theprocessor to: prior to processing the treatment image data and theplanning image data using the second deep learning engine, perform imageregistration to generate transformed image data by registering thetreatment image data against the planning image data.
 17. The system ofclaim 16, wherein the instructions for processing the treatment imagedata and the planning image data using the second deep learning engine,in response to execution by the processor, cause the processor to:process, using the second deep learning engine, the transformed imagedata and the planning image data to generate the output data.
 18. Thesystem of claim 11, wherein the instructions for processing thetreatment image data and the planning image data using the second deeplearning engine, in response to execution by the processor, cause theprocessor to: process, using a first processing pathway of the seconddeep learning engine, the treatment image data and the planning imagedata to generate first feature data associated with a first resolutionlevel; and process, using a second processing pathway of the second deeplearning engine, the treatment image data and the planning image data togenerate second feature data associated with a second resolution level;process, using a third processing pathway of the second deep learningengine, the treatment image data and the planning image data to generatethird feature data associated with a third resolution level; andgenerate the output data based on the first feature data, second featuredata and third feature data.
 19. The system of claim 11, wherein theinstructions obtaining the treatment image data and the planning imagedata, in response to execution by the processor, cause the processor toperform one of the following: obtaining the treatment image data in theform of cone beam computed tomography (CBCT) image data, and theplanning image data in the form of computed tomography (CT) image data,ultrasound image data, magnetic resonance imaging (MRI) image data,positron emission tomography (PET) image data, single photon emissioncomputed tomography (SPECT) or camera image data; obtaining thetreatment image data in the form of CT image data, and the planningimage data in the form of CT image data associated with a differentenergy level, ultrasound image data, MRI image data, PET image data,SPECT image data or camera image data; obtaining the treatment imagedata in the form of MRI image data, and the planning image data in theform of CT image data, CBCT image data, ultrasound image data, MRI imagedata, PET image data, SPECT image data or camera image data; obtainingthe treatment image data in the form of ultrasound image data, and theplanning image data in the form of CT image data, CBCT image data, PETimage data, MRI image data, SPECT image data or camera image data; andobtaining the treatment image data in the form of PET image data, andthe planning image data in the form of CT image data, CBCT image data,ultrasound image data, MRI image data, SPECT image data or camera imagedata.
 20. The system of claim 11, wherein the non-transitorycomputer-readable medium having stored thereon additional instructionsthat, in response to execution by the processor, cause the processor to:train the first or second deep learning engine to perform one of thefollowing using training data associated with past patients: automaticsegmentation to generate the output data in the form of structure dataassociated with the patient, dose prediction to generate the output datain the form of dose data associated with the patient, and treatmentdelivery data estimation to generate the output data in the form oftreatment delivery data for a treatment delivery system.